Bayesian Deep Q-Learning via Continuous-Time Flows

Efficient exploration in reinforcement learning (RL) can be achieved by incorporating uncertainty into model predictions. Bayesian deep Q-learning provides a principle way for this by modeling Q-values as probability distributions. We propose an efficient algorithm for Bayesian deep Q-learning by posterior sampling actions in the Q-function via continuous-time flows (CTFs), achieving efficient exploration without explicit assumptions on the forms of posterior distributions. Specifically, based on the recently proposed soft Q-learning framework, our algorithm learns an energy-based policy, which is distiled into a sampling network via CTF for efficient action generation. The distillation procedure relies on the recently developed technique of particle optimization in CTF. Experiments on toy and real tasks demonstrate excellent exploration ability of our algorithm, obtaining improved performance, compared to related techniques such as Stein variational gradient descent in standard soft Q-learning.